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BACKGROUND 

1 . Technical Field 

5 The invention relates to electronic devices. In particular, the invention relates to 

digital cameras and image processing used therewith. 

2. Description of Related Art 

Popularity and use of digital cameras has increased in recent years as prices 
have fallen and image quality has improved. Among other things, digital cameras 

10 provide a user or photographer with an essentially instantly viewable photographic 
image. In particular, using a built-in display unit available on most digital cameras, 
the photographer may view a photograph or image taken by the camera immediately 
after the image is captured. Moreover, digital cameras generally capture and store 
images in a native digital format. The use of a native digital format facilitates 

15 distribution and other uses of the images following an upload of the images from the 
digital camera to an archival storage/image processing system such as a personal 
computer (PC). 

While offering convenience and an ability to produce relatively high quality 
images, digital cameras are generally no less immune to various photographic 
20 inconveniences than a conventional film-based camera. For example, when taking a 
group photograph in the absence of a tripod or a willing passerby, a member of the 
group acting as the photographer is generally left out of the group picture. Similarly, 
many instances exist where one or more foreground objects partially block a view of a 
desired background scene. 

25 Accordingly, it would be desirable to have a digital camera that could alleviate 

or even overcome such photographic inconveniences. Such a digital camera would 
solve a long-standing need in the area of digital photography. 
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BRIEF SUMMARY 

In an embodiment, a method of removing an imaged object from an image using 
a digital camera is provided. The method of imaged object removal comprises 
processing within the digital camera a set of one or more captured images, a captured 
5 image of the set having an imaged object that is undesired. Processing produces a 
desired image absent the undesired imaged object. 

In another embodiment, a method of adding an imaged object to an image using 
a digital camera is provided. In another embodiment, a digital camera that produces a 
desired image from captured images is provided. 

10 Certain embodiments have other features in addition to and in lieu of the 

features described hereinabove. These and other features are detailed below with 
reference to the following drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The various features of embodiments of the present invention may be more 
15 readily understood with reference to the following detailed description taken in 
conjunction with the accompanying drawings, where like reference numerals 
designate like structural elements, and in which: 

Figure 1 illustrates a flow chart of a method of removing an imaged object from 
an image using a digital camera according to an embodiment of the present invention. 

20 Figure 2 illustrates sketched images representing exemplary images captured by 

a digital camera to depict an example of processing images according to an 
embodiment of the method of Figure 1 . 

Figure 3 illustrates sketched images representing exemplary images captured by 
a digital camera to depict another example of processing according to an embodiment 
25 of the method of Figure 1 . 

Figure 4 illustrates a flow chart of a method of adding an imaged object to a 
background image using a digital camera according to an embodiment of the present 
invention. 
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Figure 5 illustrates sketched images representing exemplary images captured by 
a digital camera to depict an example of combining images that produces a desired 
image according to an embodiment of the method of Figure 4. 

Figure 6 illustrates a block diagram of an embodiment of a digital camera that 
5 produces a desired image from a captured image according to an embodiment of the 
present invention. 

Figure 7 illustrates a backside perspective view of an embodiment of a digital 
camera that produces a desired image from a captured image according to an 
embodiment of the present invention. 

10 Figure 8 illustrates a flow chart of a method of producing a desired image from a 

captured image with a digital camera according to an embodiment of the present 
invention. 

DETAILED DESCRIPTION 

A 'desired' image is produced with a digital camera wherein the desired image 
15 is created from one or more images having undesirable characteristics when initially 
captured by the digital camera. In particular, objects or portions thereof are 
selectively added and/or removed from an image captured by the digital camera to 
produce the desired image. Moreover, the selective addition and/or removal of 
objects is performed within the digital camera as opposed to in a post-processing 
20 computer system, such as a personal computer (PC), following uploading of the 
images from the digital camera. As such, the desired image may be produced and 
stored in a memory of the digital camera in a manner that is essentially concomitant 
with capturing the images in the first place. In addition, a camera user need not wait 
until the captured images are uploaded to a PC to create and/or view the desired 
25 image. 

For example, an unwanted imaged object in a scene captured by the digital 
camera may be removed to produce a desired image of the scene without the 
unwanted imaged object, according to some embodiments. In another example, a 
flawed object from or a flawed image portion of an image captured by the digital 
30 camera may be replaced by an unflawed object from, or an unflawed image portion 
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of, another captured image. In yet another example, an object from a first image 
captured by the digital camera may be selectively added to a second captured image to 
produce the desired image, according to other embodiments. In still other 
embodiments, both image object removal and addition by the digital camera are 
5 achieved. 

Embodiments described herein provide object addition and/or removal that 
occur entirely within the digital camera. As such, a need for storing multiple 
undesirable images and/or a need for post image processing, especially using 
equipment other than the digital camera, to generate the desired image is reduced, and 
10 may be reduced or eliminated according to some embodiments. 

Figure 1 illustrates a flow chart of a method 100 of removing an imaged object 
from an image using a digital camera according to an embodiment of the present 
invention. The method 100 of imaged object removal enables selective removal of 
the imaged object from the image produced or captured by the digital camera. 

15 As used herein, 'object' generally refers to one or more of a physical object in a 

scene and a portion of a scene that may or may not include one or more physical 
objects. Additionally, an 'object' may refer to a part or portion of another physical 
object. An 'imaged object' refers to an object imaged or captured by the digital 
camera. Thus, the 'imaged object' is an object that is part of the captured image and 

20 is within a frame or boundary of the captured image. Depending on the embodiment, 
imaged object removal removes an unwanted or undesired imaged object or removes 
and then replaces the undesired image object with another, desired imaged object. 

For example, the imaged object may be a foreground object (e.g., a person) that 
partially obscures a background scene (e.g., a mountain vista). In this example, the 

25 desired image is an image of the background scene minus the imaged object. Thus for 
example, a person walking past the camera may represent an undesired or unwanted 
imaged object. According to the exemplary embodiment, the image of the person 
(i.e., undesired imaged object) is removed from the captured image to reveal an 
unobstructed image of the background scene (i.e., desired image). In addition, the 

30 method 100 of image object removal occurs within the digital camera. 
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In another example, the undesired imaged object may be eyes of a person being 
photographed where the person's eyes are closed. The desired image is a photograph 
of the person with their eyes open. The method 100 is employed to remove the 
person's closed eyes (i.e., undesired imaged object) and replace the closed eyes with 
5 an image of their open eyes. Thus, an embodiment of the method 100 may be viewed 
as removing a flawed object (e.g., closed eyes) from the image and replacing the 
flawed object with an unflawed object (e.g., open eyes). 

In yet another example, a portion of the desired image may be partially or totally 
obscured or otherwise rendered undesirable by glare or another optical artifact in the 

10 image as captured by the digital camera. In other words, the obscured portion 

represents a flawed portion of the overall image. In such instances, the undesired 
imaged object is the flawed portion of the scene containing the artifact while the 
desired image is the scene without the artifact. According to an embodiment of the 
method 100, the flawed portion of the scene containing the artifact is removed and 

15 replaced by a corresponding unflawed portion of the scene (i.e., the portion without 
the artifact) to create the desired image. 

Referring again to the flow chart illustrated in Figure 1, the method 100 of 
imaged object removal comprises capturing 110 a plurality of images using the digital 
camera. For example, capturing 110 the plurality of images may comprise capturing 

20 110 a sequence or series (i.e., set) of images the images in the series being related to 
one another. In other embodiments, the plurality of images are independent images 
and not related to one another. Capturing 110 the series may be implemented as 
either a manually captured 110 series or an automatically captured 110 series, 
depending on the embodiment of the method 100. The captured 110 series need not 

25 be time sequential. In particular, in some embodiments considerable time on the 
order of minutes or even hours may elapse between capturing 110 of individual 
images in the plurality. In yet other embodiments, capturing 110 may be capturing 
110 a single captured image. 

For example, a manually captured 1 10 series of images may be implemented by 
30 a user of the camera pressing a trigger or shutter button on the digital camera several 
times in a periodic or an aperiodic fashion. Each time the shutter button is depressed, 
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a single image of the series is captured 110. An automatically captured 110 series of 
images may be implemented as a sequence of captured 110 images that occurs at a 
predetermined rate or period when a user of the camera depresses the shutter button a 
single time. A number or quantity of images and a timing or interval of the captured 
5 images in the sequence may be programmable by a user of the camera or may be 
predetermined by a manufacturer of the digital camera. 

By way of example and not by limitation, when the user depresses the shutter 
button, a quantity of 'five' images, for example, at intervals of 'one second', for 
example, may be captured 110 automatically. Whether capturing 1 10 is manual or 
10 automatic, the series of images are captured 110 while a constant orientation of the 
camera with respect to the desired scene is maintained. By 'constant' it is meant that 
the camera orientation either does not change or does change only by an amount such 
that the essence of the scene is maintained. 

The method of removing 100 further comprises processing 120 the captured 
15 image or images within the camera to produce a desired image from which an 

undesired imaged object has been removed. With respect to a captured 110 plurality 
of images, processing 120 essentially combines or merges captured images and/or 
portions of the captured images. As a result of combining or merging, the desired 
image, which is absent the undesired imaged object, is produced. 

20 In some embodiments, processing 120 comprises removing a portion of the first 

captured image containing the undesired imaged object and recreating or replacing the 
removed image portion of the first captured image with a portion of a background 
scene of the desired image from a second captured image of the plurality. The 
background scene portion essentially is that which was originally obscured by the 

25 undesired imaged object (i.e., imaged object being removed). The portion of the 
desired image representing the originally obscured background scene portion in the 
first captured image is filled in using a corresponding image portion taken or copied 
from the second captured image of the plurality. The corresponding image portion is 
a portion of the second captured image substantially corresponding to a location and 

30 size of the removed image portion. In addition, the background scene within the 
corresponding image portion is not obscured by the undesired object in the second 



-6- 



PDNO 100205072-1 

captured image of the plurality. In various embodiments, the corresponding image 
portion from the second captured image is substituted for, overlaid onto, filled in, or 
pasted over the image portion being removed from the first captured image. Thus, by 
replacing the obscured portion of the background scene, processing 120 selectively 
5 removes the undesired imaged object from the image to produce the desired image. 

For example, the corresponding image portion may be copied or cut from the 
second captured image and used to fill in a void left in the first captured image 
resulting from removing or deleting the image portion containing the undesired 
imaged object. In another example, the corresponding image portion may be pasted 
10 over the undesired imaged object to both remove and replace the undesired imaged 
object in a single operation. 

In some embodiments, a single captured image of the plurality having a 
corresponding portion in the background scene that is entirely unobstructed by the 
undesired imaged object being removed is not available. In such cases, the 

15 corresponding image portion may be constructed or assembled from corresponding 
image portions of more than one other captured image of the plurality. Each of the 
respective corresponding image portions provides part of the unobstructed 
background scene. When assembled, the respective corresponding image portions 
yield a complete background scene corresponding to the removed portion of the first 

20 captured image. In such embodiments, the assembled corresponding image portion 
may be employed in a manner similar to that previously described hereinabove. 

Figure 2 illustrates sketched images representing exemplary images captured by 
a digital camera to depict an example of processing 120 images that combines 
portions of images according to an embodiment of the method 100. As illustrated in 
25 Figure 2, a background scene in a pair 122, 124 of captured 110 images is partially 
obscured by a person walking in a foreground of the scene. Moreover in the example 
illustrated in Figure 2, the person in each of the captured 110 images of the pair 112, 
124 obscures a different portion of the background scene. An image of the 
background scene is the desired image in the example. 

30 According to the method 100 of image object removal, an image portion 121, 

including the imaged person, is identified in a first image 122 of the pair. For 
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example, a window may be established in the first image 122, wherein the window 
encompasses or frames the image portion 121 . A rectangular window frame indicated 
by a dashed line is illustrated in Figure 2 by way of example. Other techniques to 
identify the image portion 121 include, but are not limited to, edge detection/linking 
5 and various moving target techniques known in the art. In this example, the image 
portion 121, including the imaged person, is the undesired image portion to be 
removed. 

Edge detection and edge linking techniques typically employ so-called 'gradient 
operators' to process an image. Edge linking methods generally attempt to link 

10 together multiple detected edges into a recognizable or identifiable object or shape. 
Moving target techniques generally employ statistical information sometimes 
including edge detection-based information gathered from a plurality of images to 
identify objects by virtue of a motion of an object from one image to another. 
Discussions of edge detection, edge linking, and moving target techniques are found 

15 in many image processing textbooks, including, but not limited to, Anil K. Jain, 
Fundamentals of Digital Image Processing, Prentice Hall, Inc., 1989, incorporated 
herein by reference. 

An image portion 123 in a second image 124 of the pair corresponding to the 
identified image portion 121 of the first image 122 is similarly identified. The 

20 corresponding image portion 123 of the second image 124 is then used to replace the 
image portion 121 of the first image 122 to produce a combined image 126 
representing the desired image. As illustrated in Figure 2, the image portion 121 is 
deleted or removed from the first image 122, as illustrated by portion 125. The 
corresponding image portion 123 is then copied from the second image 124 and 

25 inserted or 'pasted' into the first image 122 in place of the deleted portion 125. Once 
the corresponding image portion 123 has been pasted into the first image 122, the 
combined image 126 represents the desired image of the background scene in the 
example illustrated in Figure 2. Specifically, the combined image 126 is the desired 
image of the background scene without the person walking in the foreground. It 

30 should be noted that the image portion of the walking person in the second image 124 
alternatively could be removed and replaced by a corresponding scene portion in the 
first image 122, and still be within the scope of the present method 100. 
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In other embodiments, processing 120 comprises removing an undesired or 
flawed object or flawed image portion (i.e., object being removed) from the first 
captured image and replacing the removed flawed portion with an unflawed portion 
from a second captured image of the plurality. The flawed portion is a portion of the 
5 first captured image that contains a flaw or other undesired optical artifact. The 

unflawed portion is provided by the second captured image of the plurality. In some 
embodiments, the unflawed portion may be constructed or assembled from respective 
portions of more than one other captured image of the plurality. 

The unflawed portion replaces the flawed portion by being substituted for, 
10 overlaid onto, filled in or pasted over the flawed portion. Thus, the flawed portion 
may be deleted from the first captured image prior to being replaced by the unflawed 
portion or the unflawed portion may be essentially placed 'on top' of the flawed 
portion to replace the flawed portion in a single action. Either way, by replacing the 
flawed portion with an unflawed portion, processing 120 selectively removes the 
15 undesired object from the image to produce the desired image. 

Figure 3 illustrates sketched images representing exemplary images captured by 
a digital camera to depict another example of processing 120 images that removes and 
replaces a flawed portion of a captured image according to an embodiment of the 
method 100. As illustrated in Figure 3, a scene in a pair 122', 124' of captured 
20 images is a portrait of two people. In the example, a first image 122' includes a first 
imaged person having closed eyes, while a second image 124' includes a second 
imaged person having closed eyes. A portrait of the two people in which both people 
have open eyes is the desired image in the example. 

According to the method 100 of image object removal, an image portion 121', 
25 including the closed eyes of the first imaged person and representing the flawed 
portion, is identified in the first image 122'. For example, a window may be 
established in the first image 122', wherein the window encompasses or frames the 
image portion 121 '. A rectangular window frame indicated by a dashed line is 
illustrated in Figure 3 by way of example. In the example; the image portion 121 ', 
30 including the closed eyes of first imaged person, is the undesired image portion or 
undesired image object to be removed. 
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An image portion 123' in the second image 124' corresponding to the identified 
image portion 121' of the first image 122' is similarly identified. The corresponding 
image portion 123' of the second image 124' is used to replace the image portion 121' 
of the first image 122' to create a combined image 126' representing the desired 
5 image. Specifically, the combined image 126' is a portrait of the two people in which 
both people have open eyes in this example. 

As illustrated in Figure 3 by way of example, the image portion 121' is deleted 
or removed from the first image 122', as illustrated by portion 125'. The 
corresponding image portion 123' is then copied from the second image 124' and 

10 inserted or 'pasted' into the first image 122' in place of the deleted portion 125'. 

Once the corresponding image portion 123' has been pasted into the first image 122', 
the combined image 126' represents the desired image of the portrait scene in the 
example illustrated in Figure 3. It should be noted that the image portion of the 
closed eyes of the second imaged person in the second image 124' alternatively could 

15 be removed and replaced by a corresponding scene portion in the first image 122', 
and still be within the scope of the present method 100. 

In both of the above-described examples, cutting, deleting, or removing a 
portion of an image (e.g., image portion 121, 121') maybe accomplished by resetting 
pixels of the image corresponding to those within the portion. Inserting or pasting of 
20 a corresponding portion (e.g., corresponding image portion 123, 123') may be 

accomplished by copying pixel values from the corresponding portion into the pixels 
of the deleted portion. Alternatively, cutting and pasting may be accomplished in a 
single action by simply replacing pixel values of the deleted portion with pixel values 
of the corresponding portion. 

25 In another example (not illustrated), processing 120 compares each of the 

captured images of the plurality. During the comparison, changes from one image to 
another are detected. Processing 120 then constructs a combined image by collecting 
or assembling one or more portions of images of the plurality of captured images that 
do not contain detected changes. Image portions that do contain detected changes in 

30 one or more of the captured images are then filled in using corresponding image 

portions from a subset of the captured images in which no change was detected for the 
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image portion containing the detected change. The comparison may be performed on 
a pixel-by-pixel basis or for groups or blocks of pixels, depending on the 
embodiment. 

For example, consider a plurality of captured 110 images including five images. 
5 Further consider a first portion of the five images that remains constant across each of 
the five images, a second portion of the five images that changes from a first image to 
a second image and then remains unchanged from the second to the third image and 
so on, and a third portion that is unchanged in the first, second, and third images but 
changes in a fourth and a fifth image of the five images. 

10 In the example, processing 120 compares the exemplary five images and 

identifies the first, second, and third portions based on detected change or lack thereof 
from image to image. The combined image is then assembled by initially inserting 
the first portion into the combined image. The second portion of the combined image 
is added by copying the second portion from one or more of the second, third, fourth, 

15 and fifth image into the combined image. The third portion is then added by copying 
into the combined image the third portion from one or more of the first image, second 
and third image. Thus, the combined image produced by processing 120 includes 
those respective image portions of the five images that remain relatively constant in a 
majority of the five images. Any so-called 'moving objects' responsible for the 

20 changes detected in the five images in the example are effectively removed by such 
comparison and assembly-based processing 120. 

In yet another example (not illustrated), processing 120 is employed to remove 
flawed portions from the captured 110 image and replace the flawed portions with 
unflawed portions in other captured 110 images. In the example, flawed portions are 

25 regions of the image that include a glare or another optical artifact that detracts from 
the desirability of the image. Glare may be detected by comparing relative light 
levels between pixels or blocks of pixels in an image. Alternatively, glare may be 
detected by comparing relative light levels of a given pixel to that of an average of a 
group of pixels of the image. Color saturation with no discernable detail may be used 

30 in addition to or instead of relative light levels to detect glare, for example. The 

flawed portions containing a detected glare area are then removed and replaced with 
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corresponding portions from other captured 110 images without glare at least in the 
corresponding portions. 

Furthermore with respect to any of the above-described examples, the 
corresponding image portion(s) or constituent pixel(s) thereof may be adjusted for 
5 color saturation/hue and/or relative light level to better match the image into which 
the image portion(s) are being pasted. In addition, an overall adjustment of color 
saturation/hue, relative light level and/or image sharpness may be performed on the 
desired image prior to and/or following pasting of the portion(s). 

In other embodiments, objects, including stationary imaged objects, may be 
10 removed by processing 120 using various techniques including, but not limited to, 
parallax comparisons, inpainting, and various other image interpolation approaches. 
In parallax comparisons, several images are captured from a number of different 
positions relative to a particular, foreground stationary object to be removed, for 
example. The images are compared using the background scene or portions thereof as 
15 a frame of reference. The apparent parallax-related 'motion' of the undesired 

foreground stationary object is then employed to identify and remove the foreground 
stationary imaged object from the image. For example, parallax-related motion of the 
foreground stationary imaged object may be employed in a manner similar to that 
described hereinabove with respect to the so-called 'moving objects' to remove the 
20 stationary foreground object. 

Other techniques also may be employed instead of or in addition to those 
described hereinabove for processing 120 to remove unwanted imaged objects. For 
example in some embodiments, the above-mentioned 'image inpainting' may be used 
in processing 120 of the method 100. Georgiev et al., U. S. Pat. No. 6,587,592 Bl, 

25 incorporated herein by reference, disclose an example of image inpainting that may be 
adapted to be performed within the digital camera as the processing 120 according to 
an embodiment of the method 100. Additional information on inpainting is provided 
by C. Ballester et al., "Filling-in by Joint Interpolation of Vector Fields and Gray 
Levels", IEEE Trans. Image Process., 10 (2001), pp. 1200-121 1 ; by M. Bertalmio et 

30 al., "Image inpainting", Computer Graphics, SIGGRAPH 2000, July 2000, pp. 417- 
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424; and by Guillemo Sapiro, "Image Inpainting," SIAMNews, Volume 35, No. 4, pp. 
1-2, all three of which are incorporated by reference herein. 

Another example technique that can be adapted for processing 120 within the 
digital camera according to an embodiment of the method 100 of imaged object 
5 removal is described by Anil Korkoram et al., "A Bayesian Framework for Recursive 
Object Removal in Movie Post-Production," International Conference on Image 
Processing 2003, Barcelona, Spain, incorporated herein by reference. Korkoram et al. 
disclose a technique that employs estimation of motion based on a notion of temporal 
motion smoothness to reconstruct missing image data obscured by an unwanted object 
10 in the foreground. Korkoram et al. essentially disclose an interpolation technique for 
producing a desired image from one or more images having an unwanted moving 
object in the foreground. While intended for digital post-production processing, the 
technique of Korkoram et al. is readily adaptable to some embodiments of processing 
120. 

15 The method 100 of imaged object removal further comprises storing 130 the 

desired image in a memory of the digital camera. In particular, the combined image 
produced by processing 120 that represents the desired image is stored 130 in the 
memory of the digital camera. Thus, the plurality of captured 110 images are retained 
only temporarily until processing 120 is completed and the desired image is produced. 

20 The desired image is retained (i.e., stored 130) in memory for future viewing and is 
available for uploading to an archival image storage system, such as in a personal 
computer (PC), a microprocessor, a file server, a network disk drive, an internet file 
storage site and any other means for storing that stores archival images, such as an 
image archival storage device. 

25 The desired image produced by processing 120 may be stored 130 in one or 

more of internal memory and removable memory of the digital camera. Typically, the 
desired image is stored 130 until the desired image is uploaded to the archival image 
storage system. The desired image may be stored 130 until the desired image is 
uploaded for printing or electronic distribution by email over the Internet, for 

30 example. 
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Since only the desired image is stored 130, memory space in the digital camera 
is extended or preserved when compared to storing the plurality of images for post- 
processing as may be done conventionally. Thus, the digital camera employing the 
method 100 of imaged object removal enables the camera user or photographer to 
5 ultimately produce more desired images without needing to upload captured images 
or change the removable memory to create more storage space when compared to 
conventional post processing methods of desired image production (i.e., other than 
using the digital camera for post processing). 

Figure 4 illustrates a flow chart of an embodiment of a method 200 of adding an 
10 imaged object to an image using a digital camera according to an embodiment of the 
present invention. The method 200 of imaged object addition enables selectively 
adding an imaged object from a first image to a second image produced or captured 
by the digital camera. In an embodiment, the imaged object being added to the 
second image is an object that is part of the first image and is within a frame of the 
15 first image. 

For example, the imaged object may be a foreground object (e.g., a person) in 
the first image. The second image may be an image of a background scene, an image 
of one or more foreground objects, or an image of a background scene and one or 
more foreground objects (e.g., a group of people posing in front of a mountain vista). 
20 In this example, the 'desired' image is a combination of the foreground object of the 
first image and the background scene, foreground objects, or background scene and 
foreground objects of the second image (e.g., a combination of the person and the 
group). The method 200 of image object addition is performed within the digital 
camera. 

25 Thus according to method 200, a member of a group designated to act as a 

photographer captures an image (i.e., the second image) of the group. At different 
time, another image (i.e., the first image) of the photographer is captured. Employing 
the method 200 of image object addition, the image of the photographer (i.e., imaged 
object) is added to the second image of the group from the first image of the 

30 photographer. Thus, a combined image is produced that is an image of a complete 
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group including the group member designated to be the photographer. The combined 
image of the complete group is the desired image in the example. 

The method 200 of adding an imaged object to an image using a digital camera 
comprises capturing 210 a plurality of images with the digital camera. One or more 
5 of the captured 210 images contains an image scene and at least one of the captured 
210 images contains the imaged object to be added to the image scene. 

The method further comprises selectively combining 220 the plurality of images 
to produce a desired image. In particular, one or more imaged objects from the 
plurality of images are combined 220 with the image containing the scene. The 
10 combined 220 images become the desired image. 

For example, a first image of the captured 210 plurality may be that of a 
background scene. A second image of the captured 210 plurality may be an image of 
a first object in front of the background scene. A third image of the captured 210 
plurality may be an image of a second object in front of the background scene. Thus, 
15 the captured 210 plurality comprises the background scene image and two images 
containing separate imaged objects in front of the background scene. 

The second image and the third image may be combined 220 with the 
background scene image using a feature or features of the background scene in each 
of the images as a point or frame of reference. As such, combining 220 the images 
20 essentially collects together the first object, the second object and the background 
scene in a single desired image. 

In another example of selectively combining 220, the imaged object in the 
second image is identified and extracted from the second image. The extracted 
imaged object or image portion is then layered or inserted into the background scene 
25 image, such as a foreground object. The imaged object of the third image is similarly 
identified and extracted from the third image. The extracted imaged object from the 
third image may also be layered into the background scene image as another 
foreground object. 

Identification of the imaged object may be performed using a window, using 
30 edge detection, or another similar object identification technique. As such, the 
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imaged object may be represented in terms of an image portion containing the imaged 
object. Extraction is essentially 'cutting 5 the identified imaged object from the 
respective image using image processing. For example, cutting may be performed by 
copying only those pixels from the respective image that lie within a boundary of the 
5 identified imaged object or a window enclosing the object (e.g., image portion). 

Layering the extracted object is essentially 'pasting' the object into or in front of 
the background image. For example, pasting may be performed by replacing 
appropriate ones of pixels in the background scene image with pixels of the extracted 
object. Background scene features may be employed as points of reference in 
10 locating an appropriate location within the background scene image for layering of the 
imaged object. Alternatively, a location for imaged object layering may be 
determined essentially arbitrarily to accomplish combining 220. In other words, the 
imaged object may be placed anywhere within the background scene image. 

The method 200 further comprises storing 230 the desired image in a memory of 
15 the digital camera. In particular, the desired image produced by combining 220 is 
stored 230 in the memory of the digital camera. Thus, the captured 210 plurality of 
images need be retained only temporarily until combining 220 is completed. The 
combined image is retained (i.e., stored 130) in memory for future viewing and is 
uploadable to an archival image storage such as in a personal computer (PC), as 
20 described above for storing 120 in the method 100. 

The desired image produced by combining 220 may be stored 230 in one or 
more of internal memory and removable memory of the digital camera. Typically, the 
desired image is stored 230 until the desired image is uploaded to an archival storage 
such as, but not limited to, a personal computer (PC). Alternatively, the desired 
25 image may be stored 130 until the desired image is uploaded for printing or electronic 
distribution by email over the Internet. 

Since the plurality of captured images are stored temporarily for processing and 
then optionally deleted, the method 200 can extend memory space in the digital 
camera when compared to storing the plurality of captured images for post-processing 
30 as may be done conventionally. Thus, the digital camera employing the method 200 
of imaged object addition enables the camera user or photographer to ultimately 
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produce more desired images for storage 230 without needing to upload multiple 
images or change the removable memory to create more storage space when 
compared to conventional post processing methods of desired image production (i.e., 
other than using the digital camera). 

5 Figure 5 illustrates sketched images representing exemplary images captured by 

a digital camera to depict an example of an embodiment of combining 220 images 
that produces a desired image according to an embodiment of the method 200. As 
illustrated in Figure 5, a first image 222 of a pair of images 222, 224 contains a 
background scene along with a set of foreground objects 223 (i.e., a shaded square 
10 and a shaded triangle). A second image 224 of the pair contains the background scene 
along with another foreground object 225 (i.e., a shaded circle) not found in the first 
image 222. In this example, the other foreground object 225 is to be added to the first 
image 222 to produce the desired image. 

During combining 220 of the method 200, the other foreground object 225 of 
15 the second image 224 is copied and pasted into the first image 222. As illustrated in 
Figure 5, pasting essentially replaces a portion of the first image 222 with a copied 
image of the other foreground object 225 from the second image 224. Once pasted, 
the combined image 226 contains the background scene, the set of foreground objects 
223 from the first image 222, and the other foreground object 225 from the second 
20 image 226. 

While exemplary geometric shapes are illustrated in Figure 5 for simplicity, one 
skilled in the art will readily recognized that the foreground object may be any object 
including, but not limited to, a person, such as when a group picture of a number of 
people is missing the person of the group whom takes the picture. Combining 220 
25 provides for inserting the person missing from the group picture into the picture of the 
group to ultimately create a desired picture of the complete group. Combining 220 is 
conveniently performed in the digital camera according to the method 200 of image 
object addition. The ultimately created desired picture 226 is stored 230 by the digital 
camera in memory, while the pair of images 222, 224 optionally can be deleted. 

30 Reference herein to a 'pair' of images in some above-described examples is not 

intended to limit the embodiments of the invention to using image pairs. One or more 
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images from the plurality of captured images may be used for the methods 100 and 
200, according to various embodiments thereof. 

Figure 6 illustrates a block diagram of a digital camera 300 that produces a 
desired image from a captured image according to an embodiment of the present 
5 invention. The digital camera 300 comprises a controller 3 10, an image capture 
subsystem 320, a memory subsystem 330, a user interface 340, and a computer 
program 350 stored in the memory subsystem 330 and executed by the controller 310. 
The controller 310 interfaces with and controls the operation of each of the image 
capture subsystem 320, the memory subsystem 330, and the user interface 340. 
10 Images captured by the image capture subsystem 320 are transferred to the memory 
subsystem 330 by the controller 310 and may be displayed for viewing by a user of 
the digital camera 300 on a display unit of the user interface 340. 

The controller 310 may be any sort of component or group of components 
capable of providing control and coordination of the image capture subsystem 320, 

15 memory subsystem 330, and the user interface 340. For example, in some 

embodiments, the controller 310 is a microprocessor or microcontroller. Alternatively 
in other embodiments, the controller 310 is implemented as an application specific 
integrated circuit (ASIC) or even an assemblage of discrete components. One or more 
of a digital data bus, a digital line, or analog line may provide interfacing between the 

20 controller and the image capture subsystem 320, memory subsystem 330, and the user 
interface 340. In some embodiments of the digital camera 300, a portion of the 
memory subsystem 330 may be combined with or may be part of the controller 310 
and still be within the scope of the digital camera 300. 

In an embodiment, the controller 310 comprises a microprocessor and a 
25 microcontroller. Typically, the microcontroller provides much lower power 

consumption than the microprocessor and is used to implement low power-level tasks, 
such as monitoring button presses of the user interface 340 and implementing a real- 
time clock function of the digital camera 300. The microcontroller is primarily 
responsible for controller 310 functionality that occurs while the digital camera 300 is 
30 in a 'stand-by' or a 'shut-down' mode. The microcontroller executes a simple 

computer program. In some embodiments, the simple computer program is stored as 
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firmware in read-only memory (ROM). In some embodiments, the ROM is built into 
the microcontroller. 

On the other hand, the microprocessor implements the balance of the controller- 
related functionality. In particular, the microprocessor is responsible for all of the 
5 computationally intensive tasks of the controller 310, including but not limited to, 
image formatting, file management of the file system in the memory subsystem 330, 
and digital input/output (I/O) formatting for an I/O port or ports of the user interface 
340. 

In some embodiments, the microprocessor executes a computer program 
10 generally known as an 'operating system' that is stored in the memory subsystem 330. 
Instructions of the operating system implement the control functionality of the 
controller 310 with respect to the digital camera 300. A portion of the operating 
system may be the computer program 350. Alternatively, the computer program 350 
may be separate from the operating system. 

15 The image capture subsystem 320 comprises optics and an image sensing and 

recording circuit. In some embodiments, the sensing and recording circuit comprises 
a charge coupled device (CCD) array. During operation of the digital camera 300, the 
optics project an optical image onto an image plane of the image sensing and 
recording circuit of the image capture subsystem 320. The optics may provide either 

20 variable or fixed focusing, as well as optical zoom (i.e., variable optical 

magnification) functionality. The optical image, once focused, is captured and 
digitized by the image sensing and recording circuit of the image capture subsystem 
320. 

The controller 310 controls the image capturing, the focusing and the zooming 
25 functions of the image capture subsystem 320. When the controller 310 initiates the 
action of capturing an image, the image capture subsystem 320 digitizes and records 
the image. The recorded image is transferred to and stored in the memory subsystem 
330 as an image file. The recorded image may also be displayed on a display of the 
user interface 340 for viewing by a user of the digital camera 300, as mentioned 
30 above. 
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The memory subsystem 330 comprises memory for storing digital images, as 
well as for storing the computer program 350 and operating system of the digital 
camera 300. In some embodiments, the memory subsystem 330 comprises a 
combination of non- volatile memory (such as flash memory) and volatile memory 
5 (e.g., random access memory or RAM). The non- volatile memory may be a 
combination of removable and non-removable memory and is used in some 
embodiments to store the computer program 250 and image files, while the RAM is 
used to store digital images from the image capture subsystem 320 during image 
processing. The memory subsystem 330 may also store a directory of the images 
10 and/or a directory of stored computer programs therein, including the computer 
program 350. 

The user interface 340 comprises means for user interfacing with the digital 
camera 300 that include, but are not limited to switches, buttons 342 and one or more 
displays 344. In some embodiments, the displays 344 are each a liquid crystal display 

15 (LCD). One of the LCD displays 344 provides the user with an indication of a status 
of the digital camera 300 while the other display 344 is employed by the user to view 
images captured and recorded by the image capture subsystem 320. The various 
buttons 342 of the user interface 340 provide control input for controlling the 
operation of the digital camera 300. For example, a button may serve as an 

20 'ON/OFF' switch for the camera 300. In some embodiments, the user interface 340 is 
employed by the camera user to select from and interact with various modes of the 
digital camera 300 including, but not limited to, a mode or modes associated with 
execution and operation of the computer program 350. 

The computer program 350 comprises instructions that, when executed by the 
25 processor, implement capture of one or more images by the image capture subsystem 
320. In addition, execution of the instructions also implement processing one or more 
of the captured images to produce a desired image from the captured image. In some 
embodiments, the instructions of the computer program 350 implement selectively 
removing an imaged object from a captured image to produce the desired image. 
30 Thus in some embodiments, the instructions of the computer program 350 may 

essentially implement the method 100 of imaged object removal according to any of 
the embodiments described hereinabove. 
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In other embodiments, the instructions of the computer program 350 implement 
selectively adding an imaged object from a captured image to another captured image 
to produce the desired image. For example, a captured image containing an imaged 
object and a captured image containing a background scene are combined to produce 
5 a desired image that contains both the background scene and the imaged object. Thus 
in some embodiments, the computer program 350 may essentially implement the 
method 200 of imaged object addition according to any of the embodiments described 
hereinabove. In yet other embodiments, the instructions of the computer program 350 
implement both selectively adding and selectively removing objects from captured 
10 images to produce desire images. Thus in some embodiments, the computer program 
350 may essentially implement the method 400 described below. 

Figure 7 illustrates a backside perspective view of an embodiment of a digital 
camera 300 that produces a desired image from a captured image according to an 
embodiment of the present invention. In particular, Figure 7 illustrates exemplary 

15 buttons 342 and an exemplary image viewing LCD display 344 of the user interface 
340. In some embodiments, the buttons 342 are employed by a user of the digital 
camera 300 to select an operational mode of the digital camera 300 associated with 
imaged object removal and/or imaged object addition. The buttons 342 may also be 
used to define a window around an imaged object to be added or removed, for 

20 example. The LCD display 344 is employed to view images captured by and/or 

stored in the digital camera 300. In particular, the LCD display 344 may be used to 
view selected ones of the captured images that are to be processed to add and/or 
remove imaged objects prior to producing the desired image and/or to assist in 
directing portions of the process of adding and/or removing imaged objects by the 

25 digital camera 300. 

In addition, the LCD display 344 may be used to view a desired image produced 
by selectively adding and/or removing an imaged object. The digital camera 300 can 
process captured images to produce a desired image and further can store the desired 
image in place of the processed captured images without the need to upload the 
30 captured images into a personal computer before processing. In essence, the digital 
camera 300 comprises a self-contained processing function that ultimately extends the 
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memory of the digital camera by selectively deleting captured images and retaining 
desired images. 

Figure 8 illustrates a flow chart of an embodiment of a method 400 of producing 
a desired image from a captured image with a digital camera. The method 400 of 
5 producing a desired image comprises capturing 410 a plurality of images using a 
digital camera. The method 400 further comprises processing 420 within the digital 
camera a set of captured images from the plurality to produce a desired image from 
the set. The desired image comprises selected image portions of the captured images 
from the set. The method 400 further comprises storing 430 the desired image in a 
10 memory of the digital camera. 

In some embodiments, the set of captured images comprises an image scene that 
is common to each captured image of the set. Moreover, processing 420 occurs 
within the digital camera and in various embodiments, processing 420 comprises 
combining the captured images of the set. In such embodiments, a captured image of 

15 the set has an imaged object that is undesired in the image scene. The desired image 
of the image scene is absent the undesired imaged object in these embodiments. In 
some of these embodiments, processing 420 comprises removing from the image 
scene the imaged object that is undesired. Thus, in some embodiments, processing 
420 is similar to processing 120 described hereinabove with respect to any of the 

20 embodiments of the method 100. 

In other embodiments, the set of captured images comprises a first captured 
image including an image scene, and a second captured image including an imaged 
object. In such embodiments, the desired image comprises the image scene and the 
imaged object. In some of these embodiments, processing 420 comprises adding the 
25 imaged object to the image scene. Thus, in some embodiments, processing 420 is 
similar to combining 220 described hereinabove with respect to any of the 
embodiments of the method 200. 

In yet other embodiments, processing 420 comprises both adding an imaged 
object to an image scene from the set of captured images and removing an imaged 
30 object from an image scene from the set. In such embodiments, the added imaged 

object may be added any location in the image scene. Similarly, the removed imaged 
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object may be removed from any location in the image scene. For example, an image 
of a person may be added to an image of a group of people, such as the example 
above regarding the photographer capturing an image of a group of colleagues. 
Moreover, processing 420 provides for removing a person from an image of a group 
5 of people who is not with the group. Thus in some embodiments, processing 420 
comprises both processing 120 of the method 100 and combining 220 of the method 
200 according to any above-described embodiments thereof 

Thus, there have been described a method of imaged object removal and a 
method of imaged object addition, and collectively a method of producing a desired 

10 image from a captured image, for use in conjunction with a digital camera. In 

addition, a digital camera that produces a desired image from a captured image has 
been described. It should be understood that the above-described embodiments are 
merely illustrative of some of the many specific embodiments that represent the 
principles of the present invention. Clearly, those skilled in the art can readily devise 

15 numerous other arrangements without departing from the scope of the present 
invention as defined by the following claims. 
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